Method and apparatus for refresh management of memory modules

ABSTRACT

One embodiment sets forth an interface circuit configured to manage refresh command sequences that includes a system interface adapted to receive a refresh command from a memory controller, clock frequency detection circuitry configured to determine the timing for issuing staggered refresh commands to two or more memory devices coupled to the interface circuit based on the refresh command received from the memory controller, and at least two refresh command sequence outputs configured to generate the staggered refresh commands for the two or more memory devices

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation (and claims the benefit of priorityunder 35 USC 120) of U.S. application Ser. No. 11/929,655, filed Oct.30, 2007, which is a continuation of U.S. application Ser. No.11/828,181, filed Jul 25, 2007, which is a continuation-in-part of U.S.application Ser. No. 11/584,179, filed Oct. 20, 2006, which is acontinuation of U.S. application Ser. No. 11/524,811, filed Sep. 20,2006, which claims the benefit of U.S. provisional application Ser. No.60/823,229 filed Aug. 22, 2006, which is a continuation-in-part of U.S.application Ser. No. 11/461,439, filed Jul. 31, 2006. The disclosures ofthe prior applications are considered part of (and are incorporated byreference in) the disclosure of this application.

FIELD OF THE INVENTION

Embodiments of the present invention generally relate to memory modulesand, more specifically, to methods and apparatus for refresh managementof memory modules.

DESCRIPTION OF THE RELATED ART

The storage capacity of memory systems is increasing rapidly due tovarious trends in computing, such as the introduction of 64-bitprocessors, multi-core processors, and advanced operating systems. Suchmemory systems may include one or more memory devices, such as, forexample, dynamic random access memory (DRAM) devices. The cells of atypical DRAM device can retain data for a time period ranging fromseveral seconds to tens of seconds, but to ensure that the data isproperly retained and not lost, DRAM manufacturers usually specify avery low threshold for instituting a refresh operation. Thespecification for most modern memory systems containing DRAM devices isthat the cells of the DRAM devices are refreshed once every 64milliseconds. This means that each cell in a given DRAM device must beread out to the sense amplifier and then written back into the DRAMdevice at full signal strength once every 64 milliseconds. Furthermore,for some DRAM devices, to account for the effect of higher signal lossrate at higher temperature, the refresh rate is doubled when the deviceis operating above a standard temperature, typically above 85° C.

To simplify the task of ensuring that all DRAM cells are properlyrefreshed, most DRAM devices, including double data rate (DDR) and DDR2synchronous DRAM (SDRAM) devices, have an internal refresh row addressregister that keeps track of the row identification (ID) of the lastrefreshed row. Typically, a memory controller sends a single refreshcommand to the DRAM device. Subsequently, the DRAM device increments therow ID in the refresh row address register and executes a sequence ofstandard steps (typically referred to a “row cycle”) to refresh the datacontained in DRAM cells of all rows with the appropriate row ID's in allof the banks in the DRAM device.

With the advent of higher capacity DRAM devices, there are more cells torefresh. Thus, to properly refresh all DRAM cells in a higher capacityDRAM device, either the refresh operations need to be performed morefrequently or more cells need to be refreshed with each refresh command.To simplify memory controller design, the choice made by DRAM devicemanufacturers and memory controller designers is to keep the frequencyof refresh operations the same, but refresh more DRAM cells for eachrefresh operation for the higher capacity DRAM devices. However, oneissue associated with the action of refreshing more DRAM devices foreach refresh operation in the higher capacity DRAM devices is thatlarger electrical currents may be drawn by the higher capacity DRAMdevices for each refresh operation.

As the foregoing illustrates, what is needed in the art are newtechniques for refreshing multiple memory devices in a memory system. Inparticular, higher capacity DRAM devices that must refresh a largenumber of DRAM cells for each refresh command.

SUMMARY OF THE INVENTION

One embodiment sets forth an interface circuit configured to managerefresh command sequences. The interface circuit includes a systeminterface adapted to receive a refresh command from a memory controller,clock frequency detection circuitry configured to determine the timingfor issuing staggered refresh commands to two or more memory devicescoupled to the interface circuit based on the refresh command receivedfrom the memory controller, and at least two refresh command sequenceoutputs configured to generate the staggered refresh commands for thetwo or more memory devices.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features can be understoodin detail, a more particular description of the invention, brieflysummarized above, may be had by reference to embodiments, some of whichare illustrated in the appended drawings. It is to be noted, however,that the appended drawings illustrate only typical embodiments of thisinvention and are therefore not to be considered limiting of its scope,for the invention may admit to other equally effective embodiments.

FIG. 1A illustrates a multiple memory device system, according to oneembodiment;

FIG. 1B illustrates a memory stack, according to one embodiment;

FIG. 1C illustrates a multiple memory device system, according to oneembodiment that includes both an intelligent register and a intelligentbuffer;

FIG. 2 illustrates a multiple memory device system, according to anotherembodiment;

FIG. 3 illustrates an idealized current draw as a function of time for arefresh cycle of a single memory device that executes two internalrefresh cycles for each external refresh command, according to oneembodiment;

FIG. 4A illustrates current draw as a function of time for two refreshcycles, started independently and staggered by a time period of half ofthe period of a single refresh cycle, according to another embodiment;

FIG. 4B illustrates voltage droop as a function of a stagger offset fortwo refresh cycles, according to one embodiment;

FIG. 5 illustrates the start and finish times of eight independentrefresh cycles, according to one embodiment;

FIG. 6 illustrates a configuration of eight memory devices refreshed bytwo independently controlled refresh cycles starting at times t_(ST1)and t_(ST2), respectively, according to one embodiment;

FIG. 7 illustrates a configuration of eight memory devices refreshed byfour independently controlled refresh cycles starting at times t_(ST1),t_(ST2), t_(ST3) and t_(ST4), respectively, according to anotherembodiment;

FIG. 8 illustrates a configuration of sixteen memory devices refreshedby eight independently controlled refresh cycles t_(ST1), t_(ST2),t_(ST3) and t_(ST4), t_(ST5), t_(ST6), t_(ST7) and t_(ST8),respectively, according to one embodiment;

FIG. 9 illustrates the octal configuration of the memory devices of FIG.8 implemented within the multiple memory device system of FIG. 1A,according to one embodiment;

FIG. 10A is a flowchart of method steps for configuring, calculating,and generating the timing and assertion of two or more refresh commands,according to one embodiment;

FIG. 10B depicts a series of operations for calculating refresh staggertimes for a given configuration.

FIG. 11 is a flowchart of method steps for configuring, calculating, andgenerating the timing and assertion of two or more refresh commandscontinuously and asynchronously, according to one embodiment;

FIG. 12 illustrates the interface circuit of FIG. 1A with refreshcommand outputs adapted to connect to a plurality of memory devices,such as the memory devices of FIG. 1A, according to one embodiment;

FIG. 13 is an exemplary illustration of a 72-bit ECC DIMM based uponindustry-standard DRAM devices arranged vertically into stacks andhorizontally into an array of stacks, according to one embodiment; and

FIG. 14 is a conceptual illustration of a computer platform including aninterface circuit.

DETAILED DESCRIPTION

Illustrative information will now be set forth regarding variousoptional architectures and features of different embodiments with whichthe foregoing frameworks may or may not be implemented, per the desiresof the user. It should be strongly noted that the following informationis set forth for illustrative purposes and should not be construed aslimiting in any manner. Any of the following features may be optionallyincorporated with or without the other features described.

FIG. 1A illustrates a multiple memory device system 100, according toone embodiment. As shown, the multiple memory device system 100includes, without limitation, a system device 106 coupled to aninterface circuit 102, which is, in turn, coupled to a plurality ofphysical memory devices 104A-N. The memory devices 104A-N may be anytype of memory devices. For example, in various embodiments, one or moreof the memory devices 104A, 104B, 104N may include a monolithic memorydevice. For instance, such monolithic memory device may take the form ofdynamic random access memory (DRAM). Such DRAM may take any formincluding, but not limited to synchronous (SDRAM), double data ratesynchronous (DDR DRAM, DDR2 DRAM, DDR3 DRAM, etc.), quad data rate (QDRDRAM), direct RAMBUS (DRDRAM), fast page mode (FPM DRAM), video (VDRAM),extended data out (EDO DRAM), burst EDO (BEDO DRAM), multibank (MDRAM),synchronous graphics (SDRAM), and/or any other type of DRAM. Of course,one or more of the memory devices 104A, 104B, 104N may include othertypes of memory such as magnetic random access memory (MRAM),intelligent random access memory (TRAM), distributed networkarchitecture (DNA) memory, window random access memory (WRAM), flashmemory (e.g. NAND, NOR, or others, etc.), pseudostatic random accessmemory (PSRAM), wetware memory, and/or any other type of memory devicethat meets the above definition. In some embodiments, each of the memorydevices 104A-N is a separate memory chip. For example, each may be aDDR2 DRAM.

In some embodiments, the any of the memory devices 104A-N may itself bea group of memory devices, or may be a group in the physical orientationof a stack. For example, FIG. 1B shows a memory device 130 which iscomprised of a group of DRAM memory devices 132A-132N all electricallyinterconnected to each other and an intelligent buffer 133. Inalternative embodiments, the intelligent buffer 133 may include thefunctionality of interface circuit 102. Further, the memory device 130may be included in a DIMM (dual in-line memory module) or other type ofmemory module.

The memory devices 1032A-N may be any type of memory devices.Furthermore, in some embodiments, the memory devices 104A-N may besymmetrical, meaning each has the same capacity, type, speed, etc.,while in other embodiments they may be asymmetrical. For ease ofillustration only, three such memory devices are shown, 104A, 104B, and104 N, but actual embodiments may use any plural number of memorydevices. As will be discussed below, the memory devices 104A-N mayoptionally be coupled to a memory module (not shown), such as a DIMM.

The system device 106 may be any type of system capable of requestingand/or initiating a process that results in an access of the memorydevices 104A-N. The system device 106 may include a memory controller(not shown) through which the system device 106 accesses the memorydevices 104A-N.

The interface circuit 102 may include any circuit or logic capable ofdirectly or indirectly communicating with the memory devices 104A-N,such as, for example, an interface circuit advanced memory buffer (AMB)chip or the like. The interface circuit 102 interfaces a plurality ofsignals 108 between the system device 106 and the memory devices 104A-N.The signals 108 may include, for example, data signals, address signals,control signals, clock signals, and the like. In some embodiments, allof the signals 108 communicated between the system device 106 and thememory devices 104A-N are communicated via the interface circuit 102. Inother embodiments, some other signals, shown as signals 110, arecommunicated directly between the system device 106 (or some componentthereof, such as a memory controller or an AMB) and the memory devices104A-N, without passing through the interface circuit 102. In someembodiments, the majority of signals are communicated via the interfacecircuit 102, such that L>M.

As will be explained in greater detail below, the interface circuit 102presents to the system device 106 an interface to emulate memory deviceswhich differ in some aspect from the physical memory devices 104A-N thatare actually present within system 100. The terms “emulating,”“emulated,” “emulation,” and the like are used herein to signify anytype of emulation, simulation, disguising, transforming, converting, andthe like, that results in at least one characteristic of the memorydevices 104A-N appearing to the system device 106 to be different thanthe actual, physical characteristic of the memory devices 104A-N. Forexample, the interface circuit 102 may tell the system device 106 thatthe number of emulated memory devices is different than the actualnumber of physical memory devices 104A-N. In various embodiments, theemulated characteristic may be electrical in nature, physical in nature,logical in nature, pertaining to a protocol, etc. An example of anemulated electrical characteristic might be a signal or a voltage level.An example of an emulated physical characteristic might be a number ofpins or wires, a number of signals, or a memory capacity. An example ofan emulated protocol characteristic might be timing, or a specificprotocol such as DDR3.

In the case of an emulated signal, such signal may be an address signal,a data signal, or a control signal associated with an activateoperation, pre-charge operation, write operation, mode register setoperation, refresh operation, etc. The interface circuit 102 may emulatethe number of signals, type of signals, duration of signal assertion,and so forth. In addition, the interface circuit 102 may combinemultiple signals to emulate another signal.

The interface circuit 102 may present to the system device 106 anemulated interface, for example, a DDR3 memory device, while thephysical memory devices 104A-N are, in fact, DDR2 memory devices. Theinterface circuit 102 may emulate an interface to one version of aprotocol, such as DDR2 with 3-3-3 latency timing, while the physicalmemory chips 104A-N are built to another version of the protocol, suchas DDR with 5-5-5 latency timing. The interface circuit 102 may emulatean interface to a memory having a first capacity that is different thanthe actual combined capacity of the physical memory devices 104A-N.

An emulated timing signal may relate to a chip enable or other refreshsignal. Alternatively, an emulated timing signal may relate to thelatency of, for example, a column address strobe latency (t_(CAS)), arow address to column address latency (t_(RCD)), a row precharge latency(t_(RP)), an activate to precharge latency (t_(RAS)), and so forth.

The interface circuit 102 may be operable to receive a signal 107 fromthe system device 106 and communicate the signal 107 to one or more ofthe memory devices 104A-N after a delay (which may be hidden from thesystem device 106). In one embodiment, such a delay may be fixed, whilein other embodiments, the delay may be variable. If variable, the delaymay depend on e.g. a function of the current signal or a previoussignal, a combination of signals, or the like. The delay may include acumulative delay associated with any one or more of the signals. Thedelay may result in a time shift of the signal 107 forward or backwardin time with respect to other signals. Different delays may be appliedto different signals. The interface circuit 102 may similarly beoperable to receive the signal 108 from one of the memory devices 104A-Nand communicate the signal 108 to the system device 106 after a delay.

The interface circuit 102 may take the form of, or incorporate, or beincorporated into, a register, an AMB, a buffer, or the like, and maycomply with JEDEC standards, and may have forwarding, storing, and/orbuffering capabilities.

In one embodiment, the interface circuit 102 may perform multipleoperations when a single operation is commanded by the system device106, where the timing and sequence of the multiple operations areperformed by the interface circuit 102 to the one or more of the memorydevices without the knowledge of the system device 106. One suchoperation is a refresh operation. In the situation where the refreshoperations are issued simultaneously, a large parallel load is presentedto the power supply. To alleviate this load, multiple refresh operationscould be staggered in time, thus reducing instantaneous load on thepower supply. In various embodiments, the multiple memory device system100 shown in FIG. 1A may include multiple memory devices 104A-N capableof being independently refreshed by the interface circuit 102. Theinterface circuit 102 may identify one or more of the memory devices104A-N which are capable of being refreshed independently, and performthe refresh operation on those memory devices. In yet anotherembodiment, the multiple memory device system 100 shown in FIG. 1Aincludes the memory devices 104A-N which may be physically oriented in astack, with each of the memory devices 104A-N capable to read/write asingle bit. For example, to implement an eight-bit wide memory in astack, eight one-bit wide memory devices 104A-N could be arranged in astack of eight memory devices. In such a case, it may be desirable tocontrol the refresh cycles of each of the memory devices 104A-Nindependently.

The interface circuit 102 may include one or more devices which togetherperform the emulation and related operations. In various embodiments,the interface circuit may be coupled or packaged with the memory devices104A-N, or with the system device 106 or a component thereof, orseparately. In one embodiment, the memory devices and the interfacecircuit are coupled to a DIMM. In alternative embodiments, the memorydevices 104 and/or the interface circuit 102 may be coupled to amotherboard or some other circuit board within a computing device.

FIG. 1C illustrates a multiple memory device system, according to oneembodiment. As shown, the multiple memory device system includes,without limitation, a host system device coupled to an host interfacecircuit, also known as an intelligent register circuit 102, which is, inturn, coupled to a plurality of intelligent buffer circuits 107A-107D,memory devices which is, in turn, coupled to a plurality of physicalmemory devices 104A-N.

FIG. 2 illustrates a multiple memory device system 200, according toanother embodiment. As shown, the multiple memory device system 200includes, without limitation, a system device 204 which communicatesaddress, control, and clock signals 208 and data signals 210 with amemory subsystem 201. The memory subsystem 201 includes an interfacecircuit 202, which presents the system device 204 with an emulatedinterface to emulated memory, and a plurality of physical memorydevices, which are shown as DRAM 06A-D. In one embodiment, the DRAMdevices 206A-D are stacked, and the interface circuit 202 iselectrically disposed between the DRAM devices 206A-D and the systemdevice 204. Although the embodiments described here show the stackconsisting of multiple DRAM circuits, a stack may refer to anycollection of memory devices (e.g., DRAM circuits, flash memory devices,or combinations of memory device technologies, etc.).

The interface circuit 202 may buffer signals between the system device204 and the DRAM devices 206A-D, both electrically and logically. Forexample, the interface circuit 202 may present to the system device 204an emulated interface to present the memory as though the memorycomprised a smaller number of larger capacity DRAM devices, although, inactuality, the memory subsystem 201 includes a larger number of smallercapacity DRAM devices 206A-D. In another embodiment, the interfacecircuit 202 presents to the system device 204 an emulated interface topresent the memory as though the memory were a smaller (or larger)number of larger capacity DRAM devices having more configured (or fewerconfigured) ranks, although, in actuality, the physical memory isconfigured to present a specified number of ranks. Although the FIG. 2shows four DRAM devices 206A-D, this is done for ease of illustrationonly. In other embodiments, other numbers of DRAM devices may be used.

As also shown in FIG. 2, the interface circuit 202 is coupled to sendaddress, control, and clock signals 208 to the DRAM devices 206A-D viaone or more buses. In the embodiment shown, each of the DRAM devices206A-D has its own, dedicated data path for sending and receiving datasignals 210 to and from the interface circuit 202. Also, in theembodiment shown, the DRAM devices 206A-D are physically arranged on asingle side of the interface circuit 202.

In one embodiment, the interface circuit 202 may be a part of the stackof the DRAM devices 206A-D. In other embodiments, the interface circuit202 may be the bottom-most chip in the stack or otherwise disposed in oron the stack, or may be separate from the stack.

In some embodiments, the interface circuit 202 may perform operationswhose relative timing and ordering are executed without the knowledge ofthe system device 204. One such operation is a refresh operation. Theinterface circuit 202 may identify one or more of the DRAM devices206A-D that should be refreshed concurrently when a single refreshoperation is issued by the system device 204 and perform the refreshoperation on those DRAM devices. The methods and apparatuses capable ofperforming refresh operations on a plurality of memory devices aredescribed later herein.

In general, it is desirable to manage the application of refreshoperations such that the current draw and voltage levels remain withinacceptable limits. Such limits may depend on the number and type of thememory devices being refreshed, physical design characteristics, and thecharacteristics of the system device (e.g., system devices 106, 204.)

FIG. 3 illustrates an idealized current draw as a function of time for arefresh cycle of a single memory device that executes two internalrefresh cycles for each external refresh command, according to oneembodiment. The single memory device may be, for example, one of thememory devices 104A-N described in FIG. 1A or one of the DRAM devicesdescribed in FIG. 2.

FIG. 3 also shows several time periods, in particular, t_(RAS,) andt_(RC). There is relatively less current draw during the 35 ns periodbetween 40 ns and 75 ns as compared with the 35 ns period between 5 nsand 40 ns. Thus, in the specific case of managing refresh cyclesindependently for two memory devices (or independently for two banks),the instantaneous current draw can be minimized by staggering thebeginning of the refresh cycles of the individual memory devices. Insuch an embodiment, the peak current draw for two independent, staggeredrefresh cycles of the two memory devices is reduced by starting thesecond refresh cycle at about 30 ns. However, in practical(non-idealized) systems, the optimal start time for a second or anysubsequent refresh cycle may be a function of time as well as a functionof many variables other than time.

FIG. 4A illustrates current draw as a function of time for two refreshcycles 410 and 420, started independently and staggered by a time periodof half of the period of a single refresh cycle.

FIG. 4B illustrates voltage droop on the VDD voltage supply from thenominal voltage of 1.8 volt as a function of a stagger offset for tworefresh cycles, according to one embodiment. “Stagger offset” is definedherein as the difference between the starting times of the first andsecond refresh cycles.

A curve of the voltage droop on the VDD voltage supply from the nominalvoltage of 1.8 volt as a function of the stagger offset as shown in FIG.4B can be generated from simulation models of the interconnectcomponents and the interconnect itself, or can be dynamically calculatedfrom measurements. Three distinct regions become evident in this curve:

-   -   A: A local minimum in the voltage droop on the VDD voltage        supply from the nominal voltage of 1.8 volt results when the        refreshes are staggered by an offset such that the increasing        current transient from one refresh event counters the decreasing        current transient from another refresh event. The positive slew        rate from one refresh produces destructive interference with the        negative slew rate from another refresh, thus reducing the        effective load.

B: The best case, namely when the droop is minimum, occurs when thecurrent draw profiles have almost zero overlap.

C: Once the waveforms are separated in time so that the refresh cyclesdo not overlap additional stagger spacing does not offer significantadditional relief to the power delivery system. Consequently,thereafter, the level of voltage droop on the VDD supply voltage remainsnearly constant.

As can be seen from a simple inspection, the optimal time to begin thesecond refresh cycle is at the point of minimum voltage droop (highestvoltage), point B, which in this example is at about 110 ns. Personsskilled in the art will understand that the values used in thecalculations resulting in the curve of FIG. 4B are for illustrativepurposes only, and that a large number of other curves with differentpoints of minimum voltage droop are possible, depending on thecharacteristics of the memory device, and the electrical characteristicsof the physical design of the memory subsystem.

FIG. 5 illustrates the start and finish times of eight independentrefresh cycles, according to one embodiment of the present application.The optimization of the start times of successive independent refreshcycles may be accomplished by circuit simulation (e.g., SPICE™ orHI-SPICE as sold by Cadence Design Systems) or with logic-orientedtiming analysis tools (e.g. Verilog™ as sold by Cadence Design Systems).Alternatively, the start times of the independent refresh cycles may beoptimized dynamically through implementation of a dynamic parameterextraction capability. For example, the interface circuit 202 maycontain a clock frequency detection circuit that the interface circuit202 can use to determine the optimal timing for the independent refreshcycles. In the example of FIG. 5, the first independently controlledduple of cycles 510 and 511 begins at time zero. The next independentlycontrolled duple of cycles, cycles 520 and 521, begins approximately attime 25 nS, and the next duple at approximately 37 nSec. In thisexample, current draw is reduced inasmuch as each next duple of refreshcycles does not begin until such time as the peak current draw of theprevious duple has passed. This simplified regime is for illustrativepurposes, and one skilled in the art will recognize that other regimeswould emerge depending on the characteristic shape of the current drawduring a refresh cycle.

In some embodiments, multiple instances of a memory device may beorganized to form memory words that are longer than a single instance ofthe aforementioned memory device. In such a case, it may be convenientto control the independent refresh cycles of the multiple instances ofthe memory device that form such a memory word with multipleindependently controlled memory refresh commands, with a separaterefresh command sequence corresponding to each different instance of thememory device.

FIG. 6 illustrates a configuration of eight memory devices refreshed bytwo independently controlled refresh cycles starting at times t_(ST1)and t_(ST2), respectively, according to one embodiment. The motivationfor the refresh schedule is to minimize voltage droop while completingall refresh operations with the allotted time window, as per JEDECspecifications.

As shown, the eight memory devices are organized into two DRAM stacks,and each DRAM stack is driven by two independently controllable refreshcommand sequences. The memory devices labeled ROB01[7:4], ROB01[3:0],R1B45[7:4], and R1B45[3:0] are refreshed by refresh cycle t_(ST1) whilethe remaining memory devices are refreshed by the refresh cycle t_(ST2).

FIG. 7 illustrates a configuration of eight memory devices refreshed byfour independently controlled refresh cycles starting at t_(ST1),t_(ST2), t_(ST3) and t_(ST4), respectively, according to anotherembodiment. Such a configuration is referred to herein as a “quadconfiguration,” and the stagger offsets in this configuration arereferred to as “quad-stagger.” The quad-stagger allows for fourindependent stagger times distributed over eight devices, thus spreadingout the total current draw and lowering large slews that may result fromsimultaneous activation of refresh cycles in all eight DRAM devices.

FIG. 8 illustrates a configuration of sixteen memory devices refreshedby eight independently controlled refresh cycles, according to yetanother embodiment. Such a configuration is referred to herein as an“octal configuration.” The motivation for this stagger schedule is thesame as for the previously mentioned dual and quad configurations,however in the octal configuration it is not possible to complete allrefresh operation on all eight memories within the window unless theoperations are bunched up more closely than in the quad or dual cases.

FIG. 9 illustrates the octal configuration of the memory devices of FIG.8 implemented within the multiple memory device system 100 of FIG. 1A,according to one embodiment. As previously described, the system device106 is connected to the interface circuit 102, which, in turn, isconnected to the memory devices 104A-N. As shown in FIG. 9, there arefour independently controllable refresh command sequence outputs ofblock 930. Outputs of R0 are independently controllable refresh commandsequences. Also, outputs of R1 are independently controllable refreshcommand sequences. The blocks 930, 940, implement their respectivefunctionalities using a combination of logic gates, transistors, finitestate machines, programmable logic or any technique capable of operatingon or delaying logic or analog signals.

The techniques and exemplary embodiments for how to independentlycontrol refresh command sequences to a plurality of memory devices usingan interface circuit have now been disclosed. The following describesvarious techniques for calculating the timing of assertions of therefresh command sequences.

FIG. 10A is a flowchart of method steps for configuring, calculating,and generating the timing and assertion of two or more refresh commandsequences, according to one embodiment. Although the method is describedwith respect to the system of FIG. 1A, persons skilled in the art willunderstand that any system configured to perform the method steps, inany order, is within the scope of the claims. As shown in FIG. 10A, themethod includes the steps of analyzing the connectivity of the refreshcommand sequences between the memory devices 104 A-N and the interfacecircuit 102 outputs, calculating the timing of each of the independentlycontrolled refresh command sequences, and asserting each of the refreshcommand sequences at the calculated time. In exemplary embodiments, oneor more of the steps of FIG. 10A are performed in the logic embedded inthe interface circuit 102. In another embodiment one or more of thesteps of FIG. 10A are performed in the logic embedded in the interfacecircuit 102 while any remaining steps of FIG. 10A are performed in theintelligent buffer 133.

In one embodiment, analyzing the connectivity of the refresh commandsequences between the memory devices 104A-N and the interface circuit102 outputs is performed statically, prior to applying power to thesystem device 106. Any number of characteristics of the system device106, motherboard, trace-length, capacitive loading, memory type,interface circuit output buffers, or other physical designcharacteristics, may be used in an analysis or simulation in order toanalyze or optimize the timing of the plurality of independentlycontrollable refresh command sequences.

In another embodiment, analyzing the connectivity of the refresh commandsequences between the memory devices 104A-N and the interface circuit102 outputs is performed dynamically, after applying power to the systemdevice 106. Any number of characteristics of the system device 106,motherboard, trace-length, capacitive loading, memory type, interfacecircuit output buffers, or other physical design characteristics, may beused in an analysis or simulation in order to analyze or optimize thetiming of the plurality of independently controllable refresh commandsequences.

In some embodiments of the multiple memory device system of FIG. 1A, thephysical design can have a significant impact on the current draw,voltage droop, and staggering of the multiple independently controlledrefresh command sequences. A designer of a DIMM, motherboard, or systemwould seek to minimize spikes in current draw, the resulting voltagedroop on the VDD voltage supply, and still meet the required refreshcycle time. Some rules and guidelines for the physical design of thetrace lengths and capacitance for the signals 108, and for the packagingof the memory circuits 104A-104N as related to refresh staggeringinclude:

-   -   Reduce the inductance between intelligent buffer 133 and each        memory device 132A-N, between intelligent buffer 133 and the        intelligent register 102.    -   Increase decoupling capacitance between VDD and VSS at all        levels of the PDS: PCB, BGA, substrate, wirebond, RDL and die.    -   Separate the spikes in current draw by staggering the refresh        times between multiple memory devices.

In another embodiment, configuring the connectivity of the refreshcommand sequences between the memory devices 104A-N and the interfacecircuit 102 outputs is performed periodically at times after applicationof power to the system device 106. Dynamic configuration uses ameasurement unit (e.g., element 1202 of FIG. 12) that is capable ofperforming a series of analog and logic tests on one or more of variouspins of the interface circuit 102 such that actual characteristics ofthe pin is measured and stored for use in refresh schedulingcalculations. Examples of such characteristics include, but are notlimited to timing of response at first detected voltage change, timingof response where detected voltage change crosses thelogic_(—)1/logic_(—)0 threshold value, timing of response at peakdetected voltage change, duration and amplitude of response ring,operating frequency of the interface circuit and operating frequency ofthe DRAM devices etc.

FIG. 10B shows steps of a method to be performed periodically at sometime after application of power to the system device 106. The stepsinclude determining the connectivity characteristics of the affectingcommunication of the refresh commands, determining operating conditions,including one or more temperatures, determining the configuration of thememory (e.g. size, number of ranks, memory word organization, etc.),calculating the refresh timing for initialization, and calculatingrefresh timing for the operation phase. Similarly to the method of 10A,the method of 10B may be applied repeatedly, beginning at any step, inan autonomous fashion or based on any technically feasible event, suchas a power-on reset event or the receipt of a time-multiplexed or othersignal, a logical combination of signals, a combination of signals andstored state, a command or a packet from any component of the hostsystem, including the memory controller.

In embodiments where one or more temperatures are measured, thecalculation of the refresh timing considers not only the measuredtemperatures, but also the manufacturers specifications of the DRAMs

FIG. 11 is a flowchart of method steps for analysing, calculating, andgenerating the timing and assertion of two or more refresh commandsequences continuously and asynchronously, according to one embodiment.Although the method is described with respect to the systems of FIGS.1A, 1B, 1C, and FIG. 12, persons skilled in the art will understand thatany system configured to implement the method steps in any order, iswithin the scope of the claims. As shown in FIG. 11, the method includesthe steps of continuously and asynchronously analysing the connectivityaffecting the assertion of refresh commands between the memory devices104A-N and the interface circuit 102 outputs, continuously andasynchronously calculating the timing of each of the independentlycontrolled refresh command sequences, and continuously andasynchronously scheduling the assertion of each of the refresh commandsequences at the calculated time. In one embodiment, the method steps ofFIG. 11 may be implementation in hardware. Those skilled in the art willrecognize that physical characteristics such as capacitance, resistance,inductance and temperature may vary slightly with time and duringoperation, and such variations may affect scheduling of the refreshcommands. Moreover, during operation, the assertion of refresh commandsis intended to continue on a schedule that is not in violation of anyschedule required by the DRAM manufacturer, therefore the step ofcalculating timing of refresh command sequences and may operateconcurrently with the step of asserting refresh command sequences.

FIG. 12 illustrates the interface circuit 102 of FIG. 1A with refreshcommand sequence outputs 1201 adapted to connect to a plurality ofmemory devices, such as the memory devices 104A-N of FIG. 1A, accordingto one embodiment. In this embodiment, each of a measurement unit 1202,a calculation unit 1204, and a scheduler 1206 is configured to operatecontinuously and asynchronously.

The measurement unit 1202 is configured to generate signals 1205 and tosample analog values of inputs 1203 either autonomously at some timeafter power-on or upon receiving a command from the system device 106.The measurement unit 1202 also is operable to determine theconfiguration of the memory devices 104A-N (not shown). Theconfiguration determination and measurements are communicated to thecalculation unit 1204. The calculation unit 1204 analyses themeasurements received from the measurement unit 1202 and calculates theoptimized timing for staggering the refresh command sequences, aspreviously described herein.

Understanding the use of the disclosed techniques for managing refreshcommands, there are many apparent embodiments based uponindustry-standard configurations of DRAM devices.

FIG. 13 is an exemplary illustration of a 72-bit ECC (error-correctingcode) DIMM based upon industry-standard DRAM devices 1310 arrangedvertically into stacks 1320 and horizontally into an array of stacks,according to one embodiment. As shown, the stacks of DRAM devices 1320are organized into an array of stacks of sixteen 4-bit wide DRAM devices1310 resulting in a 72-bit wide DIMM. Persons skilled in the art willunderstand that many configurations of the ECC DIMM of FIG. 13 may bepossible and envisioned. A few of the exemplary configurations arefurther described in the following paragraphs.

In another embodiment, the configuration contains N DRAM devices, eachof capacity M that—in concert with the interface circuit(s) 1470emulates one DRAM devices, each of capacity N*M. In a system with asystem device 1420 designed to interface with a DRAM device of capacityN*M, the system device will allow for a longer refresh cycle time thanit would allow to each DRAM device of capacity M. In this configuration,when a refresh command is issued by the system device to the interfacecircuit, the interface circuit will stagger N numbers of refresh cyclesto the N numbers of DRAM devices. In one optional feature, the interfacecircuit may use a user-programmable setting or a self calibratedfrequency detection circuit to compute the optimal stagger spacingbetween each of the N numbers of refresh cycles to each of the N numbersof DRAM devices. The result of the computation is minimized voltagedroop on the power delivery network and functional correctness in thatthe entire sequence of N staggered refresh events are completed withinthe refresh cycle time expected by the system device. For example, aconfiguration may contain 4 DRAM devices, each 1 gigabit in capacitythat an interface circuit may use to emulate one DRAM device that is 4gigabit in capacity. In a JEDEC compliant DDR2 memory system, thedefined refresh cycle time for the 4 gigabit device is 327.5nanoseconds, and the defined refresh cycle time for the 1 gigabit deviceis 1275 nanoseconds. In this specific example, the interface circuit maystagger refresh commands to each of the 1 gigabit DRAM devices withspacing that is carefully selected based on the operatingcharacteristics of the system, such as temperature, frequency, andvoltage levels, while still ensuring that that the entire sequence iscomplete within the 327.5 ns expected by the memory controller.

In another embodiment, the configuration contains 2*N DRAM devices, eachof capacity M that—in concert with the interface circuit(s)1470—emulates two DRAM devices, each of capacity N*M. In a system with asystem device 1420 designed to interface with a DRAM device of capacityN*M, the system device will allow for a longer refresh cycle time thanit would allow to each DRAM device of capacity M. In this configuration,when a refresh command is issued by the system device to the interfacecircuit to refresh one of the two emulated DRAM devices, the interfacecircuit will stagger N numbers of refresh cycles to the N numbers ofDRAM devices. In one optional feature when the system device issues therefresh command to the interface circuit to refresh both of the emulatedDRAM devices, the interface circuit will stagger 2*N numbers of refreshcycles to the 2*N numbers of DRAM devices to minimize voltage droop onthe power delivery network, while ensuring that the entire sequencecompletes within the allowed refresh cycle time of the single emulatedDRAM device of capacity N*M.

As can be understood from the above discussion of the several disclosedconfigurations of the embodiments of FIG. 13, there exist at least asmany refresh command sequence spacing possibilities as there arepossible configurations of DRAM memory devices on a DIMM.

The response of a memory device to one or more time-domain pulses can berepresented in the frequency domain as a spectrograph. Similarly, thepower delivery system of a motherboard has a natural frequency domainresponse. In one embodiment, the frequency domain response of the powerdelivery system is measured, and the timing of refresh command sequencefor a DIMM configuration is optimized to match the natural frequencyresponse of the power delivery subsystem. That is, the frequency domaincharacteristics between the power delivery system and the memory deviceon the DIMM are anti-correlated such that the energy of the pulse streamof refresh command sequences spread the energy of the pulse stream outover a broad spectral range. Accordingly one embodiment of a method foroptimizing memory refresh command sequences in a DIMM on a motherboardis to measure and plot the frequency domain response of the motherboardpower delivery system, measure and plot the frequency domain response ofthe memory devices, superimpose the two frequency domain plots anddefine a refresh command sequence pulse train which frequency domainresponse, when superimposed on the aforementioned plots results in aflatter frequency domain response.

FIG. 14 is a conceptual illustration of a computer platform 1400configured to implement one or more aspects of the embodiments. As anoption, the contents of FIG. 14 may be implemented in the context of thearchitecture and/or environment of the figures previously describedherein. Of course, however, such contents may be implemented in anydesired environment.

As shown, the computer platform 1400 includes, without limitation, asystem device 1420 (e.g., a motherboard), interface circuit(s) 1470, andmemory module(s) 1480 that include physical memory devices 1481 (e.g.,physical memory devices, such as the memory devices 104A-N shown in FIG.1A). In one embodiment, the memory module(s) 1480 may include DIMMs. Thephysical memory devices 1481 are connected directly to the system 1420by way of one or more sockets.

In one embodiment, the system device 1420 includes a memory controller1421 designed to the specifics of various standards, in particular thestandard defining the interfaces to JEDEC-compliant semiconductor memory(e.g., DRAM, SDRAM, DDR2, DDR3, etc.). The specifications of thesestandards address physical interconnection and logical capabilities.FIG. 14 depicts the system device 1420 further including logic forretrieval and storage of external memory attribute expectations 1422,memory interaction attributes 1423, a data processing engine 1424,various mechanisms to facilitate a user interface 1425, and the systembasic Input/Output System (BIOS) 1426.

In various embodiments, the system device 1420 may include a system BIOSprogram capable of interrogating the physical memory module 1480 (e.g.,DIMMs) as a mechanism to retrieve and store memory attributes.Furthermore, in external memory embodiments, JEDEC-compliant DIMMsinclude an EEPROM device known as a Serial Presence Detect (SPD) 1482where the DIMM's memory attributes are stored. It is through theinteraction of the system BIOS 1426 with the SPD 1482 and theinteraction of the system BIOS 1426 with the physical attributes of thephysical memory devices 1481 that the various memory attributeexpectations and memory interaction attributes become known to thesystem device 1420. Also optionally included on the memory module(s)1480 are an address register logic 1483 (e.g. JEDEC standard register,register, etc.) and data buffer(s) and logic 1484.

In various embodiments, the compute platform 1400 includes one or moreinterface circuits 1470, electrically disposed between the system device1420 and the physical memory devices 1481. The interface circuits 1470may be physically separate from the DIMM, may be placed on the memorymodule(s) 1480, or may be part of the system device 1420 (e.g.,integrated into the memory controller 1421, etc.)

Some characteristics of the interface circuit(s) 1470, in accordancewith an optional embodiment, includes several system-facing interfacessuch as, for example, a system address signal interface 1471, a systemcontrol signal interface 1472, a system dock signal interface 1473, anda system data signal interface 1474. Similarly, the interface circuit(s)1470 may include several memory-facing interfaces such as, for example,a memory address signal interface 1475, a memory control signalinterface 1476, a memory dock signal interface 1477, and a memory datasignal interface 1478.

In additional embodiments, an additional characteristic of the interfacecircuit(s) 1470 is the optional presence of one or more sub-functions ofemulation logic 1430. The emulation logic 1430 is configured to receiveand optionally store electrical signals (e.g., logic levels, commands,signals, protocol sequences, communications) from or through thesystem-facing interfaces 1471-1474 and to process those signals. Inparticular, the emulation logic 1430 may contain one or more subfunctions (e.g., power management logic 1432 and delay management logic1433) configured to manage refresh command sequencing with the physicalmemory devices 1481.

Aspects of embodiments of the invention can be implemented in hardwareor software or both, with the software being delivered as a programproduct for use with a computer system. The program(s) of the programproduct defines functions of the embodiments (including the methodsdescribed herein) and can be contained on a variety of computer-readablestorage media. Illustrative computer-readable storage media include, butare not limited to: (i) non-writable storage media (e.g., read-onlymemory devices within a computer such as CD-ROM disks readable by aCD-ROM drive) on which information is permanently stored; and (ii)writable storage media (e.g., floppy disks within a diskette drive orhard-disk drive) on which alterable information is stored. Suchcomputer-readable storage media, when carrying computer-readableinstructions that direct the functions disclosed herein, are yet furtherembodiments.

While the foregoing is directed to exemplary embodiments, other andfurther embodiments may be devised without departing from the basicscope thereof.

1. (canceled)
 2. An interface circuit, comprising: a system interfacecircuit configured to receive from a memory controller a refresh commandfor a first number of emulated dynamic random access memory (DRAM)devices; a measurement unit configured to perform measurements of one ormore pins of the interface circuit, wherein the measurements include (i)first timing response characteristics of a particular DRAM device of adifferent, second number of physical DRAM devices, and (ii) secondtiming response characteristics of a different DRAM device of the secondnumber of physical DRAM devices; a calculation unit configured to:receive information related to the first measured timing responsecharacteristics and the second measured timing response characteristicsfrom the measurement unit; and determine, based at least in part on theinformation received from the measurement unit, timing for issuing asequence of staggered refresh commands to the second number of physicalDRAM devices that appear to the memory controller as the first number ofemulated DRAM devices, wherein the sequence of staggered refreshcommands to the second number of physical DRAM devices completes in arefresh cycle time based on the refresh command from the memorycontroller for the first number of emulated DRAM devices; and ascheduler configured to order staggered refresh commands for the secondnumber of physical DRAM devices based on the timing determined by thecalculation unit
 3. The interface circuit of claim 2, wherein: themeasurement unit is further configured to determine a configuration ofthe second number of physical DRAM devices, wherein the configurationincludes one or more of size, number of ranks, or memory wordorganization associated with the second number of physical DRAM devices4. The interface circuit of claim 2, wherein the measurement unit isfurther configured to perform a series of measurements to determinetiming response characteristics of each of the second number of physicalDRAM devices.
 5. The interface circuit of claim 2, wherein themeasurement unit is further configured to perform a series ofmeasurements to determine timing responses of the particular DRAM deviceat selected logic threshold levels.
 6. The interface circuit of claim 2,wherein the measurement unit is further configured to perform a seriesof measurements to determine a timing response of the particular DRAMdevice at a peak detected voltage change, a duration and a responsering.
 7. The interface circuit of claim 2, wherein the measurement unitis further configured to perform a series of measurements to determinean operating temperature of each of the second number of physical DRAMdevices.
 8. The interface circuit of claim 2, wherein the staggeredrefresh cycles are staggered by equal amounts of time within a singlerefresh cycle associated with the refresh command received from thememory controller.
 9. The interface circuit of claim 2, wherein thecalculation unit is configured to determine N staggered refresh cyclesfor 2*N DRAM devices coupled to the interface circuit that are staggeredby an amount of time equal to approximately an N-th of a single refreshcycle associated with the refresh command received from the memorycontroller, wherein N is a number greater than one.
 10. The interfacecircuit of claim 2, wherein the calculation unit is configured todynamically determine, at a time subsequent to applying power to thesecond number of physical DRAM devices, timing for issuing a sequence ofstaggered refresh commands.
 11. The interface circuit of claim 2,wherein the system interface further comprises a memory address signalinterface, a memory control signal interface, a memory clock signalinterface, and a memory data signal interface.
 12. The interface circuitof claim 2, further comprising emulation logic configured to emulate aninterface protocol of each of the first number of emulated DRAM devices.13. The interface circuit of claim 2, wherein the second number ofphysical DRAM devices are arranged in one or more stacks, and astaggered refresh cycle is transmitted to two or more memory devicesresiding within the same stack.
 14. The interface circuit of claim 13,wherein the second number of physical DRAM devices are arranged in onestack, and the interface circuit is integrated within the one stack. 15.The interface circuit of claim 2, wherein the interface circuit and thesecond number of physical DRAM device are included in a dual in-linememory module.
 16. The interface circuit of claim 2, wherein themeasurements are analog signal measurements.
 17. The memory module ofclaim 2, wherein a time difference between a start time of each of thesequence of staggered refresh commands relative to the refresh commandfrom the memory controller differs for each of the staggered refreshcommands.
 18. A memory module comprising: a first number of physicalmemory devices; and an interface circuit comprising: a system interfacecircuit configured to receive from a memory controller a refresh commandfor a second number of emulated memory devices; a measurement unitconfigured to perform measurements of one or more pins of the interfacecircuit, wherein the measurements include (i) first timing responsecharacteristics of a particular memory device of the first number ofphysical memory devices, and (ii) second timing response characteristicsof a different memory device of the first number of physical memorydevices; a calculation unit configured to: receive information relatedto the first measured timing response characteristics and the secondmeasured timing response characteristics from the measurement unit; anddetermine, based at least in part on the information received from themeasurement unit, timing for issuing a sequence of staggered refreshcommands to the first number of physical memory devices that appear tothe memory controller as the second number of emulated devices, whereinthe sequence of staggered refresh commands to the first number ofphysical memory devices completes in a refresh cycle time based on therefresh command from the memory controller for the second number ofemulated memory devices; and a scheduler configured to order staggeredrefresh commands for the first number of physical memory devices basedon the timing determined by the calculation unit.
 19. The memory moduleof claim 18, wherein the calculation unit is configured to dynamicallydetermine, at a time subsequent to applying power to the first number ofphysical memory devices, timing for issuing a sequence of staggeredrefresh commands.
 20. A memory module comprising: 2*N physical memorydevices each having a capacity of M; and an interface circuitcomprising: a system interface circuit configured to receive from amemory controller a refresh command for two emulated memory devices,wherein each of the two emulated memory devices has a capacity of N*M; ameasurement unit configured to perform measurements of one or more pinsof the interface circuit, wherein the measurements include (i) firsttiming response characteristics of a particular DRAM device of the 2*Nphysical DRAM devices, and (ii) second timing response characteristicsof a different DRAM device of the 2*N physical DRAM devices; acalculation unit configured to: receive information related to the firstmeasured timing response characteristics and the second measured timingresponse characteristics from the measurement unit; and determine, basedat least in part on the information received from the measurement unit,timing for issuing a sequence of staggered refresh commands to the 2*Nphysical memory devices that appear to the memory controller as the twoemulated devices, wherein the sequence of staggered refresh commands tothe 2*N physical memory devices completes in a refresh cycle time basedon the refresh command from the memory controller for the two emulatedmemory devices; and a scheduler configured to order staggered refreshcommands for the 2*N physical memory devices based on the timingdetermined by the calculation unit.
 21. The memory module of claim 20,further comprising emulation logic configured to emulate an interfaceprotocol of the two emulated memory devices each having a capacity ofN*M.